﻿<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
 <title>Testing Across 12 OSes, 10 Versions of PHP and 6 Databases  - Blog - Flourish</title><link rel="start" href="http://flourishlib.com/"><link rel="search" href="http://flourishlib.com/search"><link rel="icon" href="http://static.flourishlib.com/favicon.ico" type="image/x-icon"><link rel="shortcut icon" href="http://static.flourishlib.com/favicon.ico" type="image/x-icon"><link rel="alternate" href="http://flourishlib.com/blog?format=rss" title="RSS Feed" type="application/rss+xml"><link rel="alternate" href="http://flourishlib.com/blog/60000Tests" title="Plain Text" type="text/x-trac-wiki">
  <link rel="stylesheet" href="/img/site.css" type="text/css">
  <script type="text/javascript" src="/img/trac.js"></script>
  <meta name="attribution" content="Code Repository icons from http://www.famfamfam.com/lab/icons/silk/">
  <meta name="verify-v1" content="kTg9BLsDLvfjCAvuTxcniw9+tap5BDBeN0Xs/MfOYFE=">
  <meta name="msvalidate.01" content="69478FBEB92A4D48978C7A96081FDDF8">
  <meta name="y_key" content="26345003a345314a">
  <?php echo $this->Html->css('site'); ?>
<script src="/img/in.php" async="" type="text/javascript"></script></head>
<body>


<?php 
	echo $this->Session->flash('auth'); 
?>


<div id="header">
	<a href="http://flourishlib.com/" id="logo"><img src="/img/header_logo_beta.gif" alt="Flourish"></a>
	<form action="/search" method="get">
		<div>
			<input style="color: rgb(153, 153, 153);" name="q" value="Search…" type="text">
			<input name="noquickjump" value="1" type="hidden">
			<noscript>
				<div class="button">
					<input type="submit" value="Go" />
				</div>
			</noscript>
		</div>
	</form>
	<ul><li class="first"><a accesskey="1" href="http://flourishlib.com/docs">Documentation</a></li><li><a href="http://flourishlib.com/docs/Download">Download</a></li><li><a href="http://flourishlib.com/browser">Code</a></li><li><a href="http://flourishlib.com/docs/Tests">Tests</a></li><li><a accesskey="7" href="http://flourishlib.com/newticket">Tickets</a></li><li><a href="http://flourishlib.com/discussion">Discussion</a></li><li><a accesskey="2" href="http://flourishlib.com/timeline">Project</a></li><li class="active  last"><a href="http://flourishlib.com/blog">Blog</a></li></ul>
</div>
<div id="main">




<div id="ctxtnav" class="nav">
 <h2>Wiki Navigation</h2>
 <!--
 
 -->
</div>  

<div id="content" class="wiki">

 
  
  
   
   <div>
	<div id="searchable">
		<h1 id="TestingAcross12OSes10VersionsofPHPand6Databases">Testing Across 12 OSes, 10 Versions of PHP and 6 Databases<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#TestingAcross12OSes10VersionsofPHPand6Databases"> ¶</a></h1>
<div class="secondary_info">
	<div class="aside">
		<h2 class="rss"><a href="http://feeds2.feedburner.com/flourishlib_blog"><img src="/img/rss.gif" alt="(RSS Icon)"> RSS Feed</a></h2>
	</div>
	<div class="aside blog_stats">
		<p>
			Posted by <strong>Will Bond</strong> on <strong>8/4/10</strong> at <strong>2:51 pm</strong>,
			
				<a href="#Comments">5 comments</a>
			
		</p>
	</div>
	<div class="aside">
		<a class="show_all" href="http://flourishlib.com/blog">Show all</a>
		<h2>Recent Blogs</h2>
		<ul class="recent_blogs">
		
			<li><a href="http://flourishlib.com/blog/PHPFloatingPointBug">PHP Floating Point Bug</a> <small>1/14/11</small></li>
		
			<li><a href="http://flourishlib.com/blog/60000Tests">Testing Across 12 OSes, 10 Versions of PHP and 6 Databases</a> <small>8/4/10</small></li>
		
			<li><a href="http://flourishlib.com/blog/OfflineDocumentation">Offline Documentation</a> <small>8/2/10</small></li>
		
			<li><a href="http://flourishlib.com/blog/DoYouUseODBC">Do You Use ODBC?</a> <small>6/22/10</small></li>
		
			<li><a href="http://flourishlib.com/blog/DB2SupportAdded">DB2 Support Added</a> <small>4/14/10</small></li>
		
		</ul>
	</div>
	<div class="aside">
		<h2>About</h2>
		<p>
			<a href="http://flourishlib.com/">Flourish</a> is a PHP unframework, a general-purpose,
			object-oriented library focused on security, <a href="http://flourishlib.com/docs">documentation</a>, ease-of-use and portability.
		</p>
	</div>
	<div class="aside social">
		<h2>Around the Web</h2>
		<ul class="social">
			<li>
				<a href="http://twitter.com/flourishlib">
					<img src="/img/twitter.gif" alt="(Twitter Icon)">
					Twitter
				</a>
			</li>
			<li>
				<a href="http://ohloh.net/p/flourish/">
					<img src="/img/ohloh.gif" alt="(Ohloh Icon)">
					Ohloh
				</a>
			</li>
			<li>
				<a href="http://github.com/wbond/flourish">
					<img src="/img/github.gif" alt="(GitHub Icon)">
					GitHub
				</a>
			</li>
			<li>
				<a href="http://bitbucket.org/wbond/flourish">
					<img src="/img/bitbucket.gif" alt="(Bitbucket Icon)">
					Bitbucket
				</a>
			</li>
			<li>
				<a href="http://launchpad.net/flourish">
					<img src="/img/launchpad.gif" alt="(Launchpad Icon)">
					Launchpad
				</a>
			</li>
			<li>
				<a href="http://webchat.freenode.net/?channels=#flourishlib">
					<img src="/img/irc.gif" alt="(IRC Icon)">
					IRC
				</a>
			</li>
		</ul>
	</div>
</div>
<p>
Part of the goal of Flourish is to provide a solid library of code to 
build web apps and websites on top of. Since PHP runs on such a variety 
of operating systems and many users do not control the server they are 
installed on, operating system and PHP version compatibility is 
important. Thanks to recent donations by <a class="ext-link" href="http://imarc.net/"><span class="icon">iMarc</span></a> and Luke Foreman, the <a class="wiki" href="http://flourishlib.com/docs/Tests">testing setup</a> has been expanded to include OS X 10.6 and Windows Server 2008.
</p>
<p>
With these changes, <a class="wiki" href="http://flourishlib.com/docs/Tests">over 60,000 tests</a>
 are run with every SVN commit, on a total of 10 different versions of 
PHP, running on 12 different server environments. All database-related 
tests are run on the six supported databases, and most are run using 
each of the 14 supported database extensions against eight different 
database versions. In situations where Flourish provides a pure-PHP 
fallback implementation of a non-standard PHP extension, the tests are 
run both with the non-standard extension loaded and not.
</p>
<div style="width: 600px;">
<div class="stacked_first" style="width: 35%;">
	<h3>Operating Systems</h3>
	<ul>
		<li>CentOS 5.3</li>
		<li>Debian 5.0</li>
		<li>Fedora 12</li>
		<li>FreeBSD 7.2</li>
		<li>NetBSD 5.0.1</li>
		<li>OpenBSD 4.5</li>
		<li>OpenSolaris 2009.06</li>
		<li>OpenSuSE 11.2</li>
		<li>OS X 10.6</li>
		<li>Ubuntu 8.04 LTS</li>
		<li>Windows Server 2008</li>
		<li>Windows XP</li>
	</ul>
</div>
<div class="stacked" style="width: 25%;">
	<h3>PHP Versions</h3>
	<ul>
		<li>5.3.3</li>
		<li>5.3.2</li>
		<li>5.3.1</li>
		<li>5.2.13</li>
		<li>5.2.12</li>
		<li>5.2.9</li>
		<li>5.2.8</li>
		<li>5.2.6</li>
		<li>5.2.4</li>
		<li>5.1.6</li>
	</ul>
</div>
<div class="stacked">
	<h3>Databases</h3>
	<ul>
		<li>DB2 Express-C 9.7</li>
		<li>MySQL 5.0</li>
		<li>Oracle 10g XE</li>
		<li>PostgreSQL 8.3</li>
		<li>SQLite 2</li>
		<li>SQLite 3</li>
		<li>SQL Server 2005</li>
		<li>SQL Server 2008</li>
	</ul>
</div>
<div style="clear: left;"></div>
</div>
<p>
At the end of the day, this helps to ensure that Flourish will work if 
you deploy it on a RHEL 4 server running PHP 5.1.6, a Windows Server 
2008 machine running PHP 5.3.3 or a Solaris box with PHP 5.2.9.
</p>
<h2 id="TheSetup">The Setup<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#TheSetup"> ¶</a></h2>
<p>
Throwing around numbers is great and all, but I'd like to touch on the setup a little bit. Flourish uses <a class="ext-link" href="http://phpunit.de/"><span class="icon">PHPUnit</span></a>
 for unit and integration tests. And yes, many of the test aren't true 
unit tests. Since portability and compatibility are key, I want all of 
the code to actually run. If I were to have written pure unit tests, I 
would have mocked all sorts of dependencies and missed quite a number of
 bugs in PHP and bizarre edge cases.
</p>
<div style="text-align: center; float: left; margin-right: 1.5em; font-size: 0.8em; margin-top: 1.1em; margin-bottom: 1.5em;">
    <a href="http://static.flourishlib.com/tests_example-large.png"><img src="/img/tests_example.png" alt="tests.php example output" style="border: 6px solid rgb(183, 183, 183);"></a><br><tt>tests.php</tt> example output
</div>
<p>
PHPUnit contains an awesome amount of functionality, however when I 
started with it, it did not allow for running tests with different PHP 
configuration. I wouldn't be surprised if <a class="ext-link" href="http://twitter.com/s_bergmann"><span class="icon">Sebastian</span></a>
 has added the functionality in 3.4 or 3.5, but I started this whole 
process back with 3.3 so I created my own solution. There is a PHP test 
runner <tt>tests.php</tt> that parses class config files and runs the 
tests once for each different configuration. It has evolved over time, 
and isn't the prettiest thing I've ever written, but it gets the job 
done.
</p>
<p>
<tt>tests.php</tt> includes some other functionality, including a 
column-based, color-coded shell output for interactive testing and JSON 
output for automated testing. It allows for configuration values to be 
passed at run time (through a nasty PHP ini hack) so that passwords 
don't need to be hard coded in the tests and inadvertantly leaked 
through a source code repositories. In addition, you can included or 
exclude specific classes and filter the configurations to be run.
</p>
<div style="float: right; text-align: center; font-size: 0.8em; margin-left: 1.5em; margin-top: 1.1em; margin-bottom: 1.5em;">
    <a href="http://static.flourishlib.com/remote_tests_example-large.png"><img src="/img/remote_tests_example.png" alt="remote_tests.sh example output" style="border: 6px solid rgb(183, 183, 183);"></a><br><tt>remote_tests.sh</tt> example output
</div>
<p>
Once the local test runner once was written, I went to work setting up a
 bunch of virtual machines. There are a total of 12 VMs running on two 
different physical machines, an Intel i7 quad-core with 8GB of ram and a
 dual-core iMarc with 3GB ram. One of the virtual machines runs DB2, 
MySQL, Oracle and PostgreSQL, while another runs SQL Server 2005 and 
2008. Every machine, including Windows, is running OpenSSH, allowing for
 key-based authentication and distribution of the tests. I leave the 
stock OS configuration alone as much as possible, but install PHP, 
various extensions, PHPUnit, some command line database programs and 
configure the various local sendmail implementations to route through an
 SMTP relay. Where possible I use the OS-supplied PHP packages in an 
effort to find bugs related to common environments.
</p>
<p>
To run the rests on the virtual machines, I use a custom bash script <tt>remote_tests.sh</tt> that <tt>scp</tt>s a tarball to each machine and then invokes <tt>tests.php</tt> over <tt>ssh</tt>. <tt>remote_tests.sh</tt>
 allows for parallel execution of the test scripts for the different 
database types, however I found that performance decreased due to disk 
thrashing. It also allows checking the remote server configuration to 
show the installed PHP version and the install status of both required 
and optional PHP extensions.
</p>
<p>
I haven't spent much time abstracting or refining this setup for use by 
others, but I could see it being useful for library and framework 
maintainers. You can check out the code at <a class="ext-link" href="http://github.com/wbond/flourish/tree/master/tests/"><span class="icon">http://github.com/wbond/flourish/tree/master/tests/</span></a>.
</p>
<hr>
<h2 id="Comments">Comments<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#Comments"> ¶</a></h2>
<div class="comment">
		
		<h3 id="comment_35">awake   at 4:43 pm on Aug 4, 2010<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#comment_35"> ¶</a></h3>
		<p>
awesome stuff!
</p>

	</div>

	<div class="comment">
		
		<h3 id="comment_36">awake   at 5:38 pm on Oct 14, 2010<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#comment_36"> ¶</a></h3>
		<p>
any plan for mongo db support?
</p>

	</div>

	<div class="comment">
		
		<h3 id="comment_39">ilo   at 7:32 pm on Dec 2, 2010<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#comment_39"> ¶</a></h3>
		<p>
Definitely that is the way to go, congratulations!
</p>
<p>
I've been playing a little bit with the testing stuff and there is one 
improvement I would suggest to include: modify remote testing to allow 
patching the source before start the testing. This will allow to test 
patches even before they are committed. 
</p>
<p>
From my understanding of the remote_test script (I'm missing part of the
 testing infrastructure) this can be done easily using a new command 
line flag (-f of patchFile is available = url of patch file) and trying 
to apply before creating the tarball. Testing a copy of current folder 
so patching and revert operations do not break other modifications is a 
must in this case. 
</p>
<p>
If you need help with this or want to discuss more about it count with me.
</p>

	</div>

	<div class="comment">
		
		<h3 id="comment_40">wbond   at 8:28 am on Dec 4, 2010<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#comment_40"> ¶</a></h3>
		<p>
@ilo
</p>
<p>
Using <tt>remote_tests.sh</tt> it is possible to run test a modified 
copy of Flourish. Rather than building the patching process into the 
test framework, simply patch the file via your preferred mechanism (<tt>patch</tt>, TortoiseSVN, etc) and then run the tests.
</p>
<p>
In terms of automating the testing of patches - I'd at least want to 
review a patch before running the tests, just from a security 
perspective.
</p>

	</div>

	<div class="comment">
		
		<h3 id="comment_41">ilo   at 7:08 pm on Dec 4, 2010<a title="Link to this section" class="anchor" href="http://flourishlib.com/blog/60000Tests#comment_41"> ¶</a></h3>
		<p>
@wbond, For sure, I never meant for a patch to be tested automatically. 
About being integrated with the trac (just commented in our quick irc 
conversation) I was meaning that knowing the environments where a patch 
fails would help a lot to find and debug or fix the patch. You are the 
one with enough knowledge of the testing framework behind flourishlib, 
so probably you are the only one with full understanding about the 
commit and testing processes.
</p>

	</div>


<form action="#comment_preview" method="post" id="add_comment"><div><input name="__FORM_TOKEN" value="fc5f84415a87ea6d47c5c0a1" type="hidden"></div>
	<fieldset>
		<legend>Add comment</legend>
		
		
		<div class="field">
			<div class="wikitoolbar"><a tabindex="400" title="Bold text: '''Example'''" id="strong" href="#"></a><a tabindex="400" title="Italic text: ''Example''" id="em" href="#"></a><a tabindex="400" title="Heading: == Example ==" id="heading" href="#"></a><a tabindex="400" title="Link: [http://www.example.com/ Example]" id="link" href="#"></a><a tabindex="400" title="PHP Code block: {{{ example }}}" id="php" href="#"></a><a tabindex="400" title="HTML Code block: {{{ example }}}" id="ehtml" href="#"></a><a tabindex="400" title="SQL Code block: {{{ example }}}" id="sql" href="#"></a><a tabindex="400" title="Javascript Code block: {{{ example }}}" id="js" href="#"></a><a tabindex="400" title="Code block: {{{ example }}}" id="code" href="#"></a><a tabindex="400" title="New paragraph" id="np" href="#"></a><a tabindex="400" title="Line break: [[BR]]" id="br" href="#"></a></div><textarea class="wikitext" id="comments" name="comments" cols="80" rows="5"></textarea>
		</div>
		<div class="field">
			<label for="commenter">Your name, email or username (this <em>will</em> be shown):</label><br>
			<input id="commenter" name="commenter" size="30" type="text">
		</div>
		<div class="field">
			<input size="30" value="Post Comment" type="submit">
			<input name="preview" value="Preview" type="submit">
			<input name="comment_id" value="" type="hidden">
			<input name="action" value="comment" type="hidden">
		</div>
		<script type="text/javascript" src="/img/wikitoolbar.js"></script>
	</fieldset>
</form>

	</div>
   </div>
   
  
  
  <script type="text/javascript">
   addHeadingLinks(document.getElementById("searchable"), "Link to this section");
  </script>
 
 
</div>

<script type="text/javascript">searchHighlight()</script>
<div id="altlinks"><h3>Download in other formats:</h3><ul><li class="first"><a href="http://flourishlib.com/blog?format=rss" class="rss">RSS Feed</a></li><li class="last"><a href="http://flourishlib.com/blog/60000Tests">Plain Text</a></li></ul></div>
	<div style="clear: both;"></div>
</div>



<div id="footer">
	<div id="footer_nav">
		<ul><li class="first"><a href="http://flourishlib.com/login">Login</a></li><li class="last"><a href="http://flourishlib.com/register">Register</a></li></ul>
	</div>
	<div class="copyright">
		© 2009 Will Bond
		<a class="license" href="http://flourishlib.com/license">License</a>
		<a class="donate" href="http://flourishlib.com/docs/Support">Support</a>
	</div>
	  
	  
	  
		
	  
	<div class="clear"></div>
</div>



<script type="text/javascript">
	var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
	document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script><script src="/img/ga.js" type="text/javascript"></script>
<script type="text/javascript">
	var pageTracker = _gat._getTracker("UA-688035-5");
	pageTracker._initData();
	pageTracker._trackPageview();
</script>
<script src="/img/67539.js" type="text/javascript"></script>
<noscript><p><img alt="Clicky" width="1" height="1" src="http://static.getclicky.com/67539-db9.gif" /></p></noscript>



</body></html>